migrate os.listdir() to os.scandir() to increase performance#78
Conversation
|
I did very simple and trivial benchmark on walking through the kernel tree:
|
|
If this gets merged, I will open another PR for the remaining occurrences of |
As recommended in https://peps.python.org/pep-0471/ Signed-off-by: Miroslav Suchý <msuchy@redhat.com>
|
Any review comments? |
|
@AyanSinhaMahapatra or @pombredanne can I kindly ask you for a review? |
There was a problem hiding this comment.
@xsuchy Thanks++ for the PR.
Thanks for the references, we should definitely move towards using scandir() allright in this context. I've tested out this branch with scancode-toolkit latest too, and everything looks good, and we do get some performance improvements there on the codebase collection step similarly as indicated by you, thanks for doing and including those too.
- Some more CI updates and other fixes were also added to main, could you rebase to latest main?
- Please go ahead and use scandir() in the other instances of listdir() use in commoncode too.
This looks ready otherwise, and apologies for the late review.
We might want to do the same in other places too which would have some effect on the performance, https://github.com/search?q=org%3Aaboutcode-org+os.listdir&type=code maybe this could be useful in extractcode/fetchcode/deltacode too
2 participants
As recommended in https://peps.python.org/pep-0471/